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In Memoriam: 


Hans Petter William Sirevag Selasky 


e FreeBSD community was recently saddened by the tragic death of one of its most pro- 
lif c contributors. We leamed that Hans Petter Selasky passed away in a traff c accident in 
Lillesand, Norway on June 23, 2023 at the age of 41 Hans was an incredibly brilliant and kind 
person, and made many valuable contributions to FreeBSD. He was preceded in death by his 
foaled oe and is survived by his mother, Inger Elisabeth, his brothers Mark and Leif Con- 
= =, ii rad, and his nieces and nephews Petra, David and Signe. 

Hans began contributing to FreeBSD roughly 25 years ago, 
with f xes to FreeBSD’s ISDN support. He was a FreeBSD com- 
mitter for nearly 5 years, and was best known for re-writ- 
ing and maintaining the USB stack. Hans wrote the webcamd 
package which supports running Linux webcam drivers in 
userspace on FreeBSD, and which enables those of us using 
FreeBSD on the desktop to participate in modern teleconfer- 

y  encing. Most recently, he worked for Mellanox (now NVID- 
Photo courtesy Ollivier Robert IA) to support their ConnectX series of high speed NICs on 
FreeBSD. Hans's work included major contributions to the kernel TLS framework, as well 
as support for NIC kTLS send and receive off oad in the mce(4) driver, and many improve- 
ments to the Linux device driver compatibility layer. 

| f rst met Hans in 205, in the context of his work on the mce(4) driver for Mellanox 
NICs. We worked together to make the mce(4) driver one of highest performance NIC 
drivers in FreeBSD. It was during this time that | learned how brilliant he was. He often had 
ideas that sounded “crazy,” but were actually brilliant. One example of this was his idea to 
sort incoming TCP packets using the NIC-provided RSS f ow identif ers in order to present 
LRO with all packets from the same TCP connection back to back. This idea, which | initially 
discounted as impractical, was crucial to Netf ix being able to meet our performance tar- 
get of serving 00Gh/s of video traff c from a single machine, and continues to save Net- 

f ixa large amount of CPU resources. 

Hans was a kind and welcoming person. The f rst time | attended EuroBSDCon was in 
208 in Lillehammer, Norway, where Hans insisted on playing host to me. Hans had driven 
across Norway from his home in Grimstad to EuroBSDCon in Lillehammer with his father, 
and took me around to see the Olympic ski jump and several other sites in the town. He 
then took me out to dinner and back to the house he'd rented with his father for an eve- 
ning of great conversation. 

Outside of FreeBSD, Hans'’s hobbies included music and mathematics. He was active in 
his church, and contributed to its sound team. He was a loving and dedicated uncle to his 
nieces and nephews. He loved animals, especially his cat Pumba. 

Even if you don’t use FreeBSD yourself, odds are good that Han’s work touches your 
daily life. For example, if you use a Playstation, chances are you are using Hans'’s USB stack. 
If you watch Nettf ix, the odds are good that the show you're watching was delivered to you 
by a ConnectX NIC running Hans's mce(4) driver. 

Hans, if you are reading this, Know that you will be missed. 


By Drew Gallatin 
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LETTER 


from the Foundation 


Dear Readers, 

lm excited to introduce our July/ 
August edition! For almost a decade, 
the FreeBSD Foundation has been 
producing the Journal, and we take 
pride in delivering informative and 
helpful content to BSD and Unix 
enthusiasts globally. 

Within this issue, you'll f nd 
enlightening pieces on Containers 
and Cloud, including articles on ports, Jails, and 
virtualization. Additionally, you're in for a fun read with 
Michael Lucas’ “We Get Letters” segment, which is 
a blend of Michael's wit and seriousness, making it a 
truly enjoyable read. 

Drew Gallatin wrote a beautiful tribute to long- 
term FreeBSD developer Hans Petter Selasky, who 
unexpectedly passed away in June. He will truly be 
missed in our community, not only for his expertise 
in many areas of the operating system, but for his 
kindness and support of others in the Project. 

Finally, check out the Events Calendar, which 
highlights upcoming events that might be of interest 
to folks in our community. 

Thank you for your ongoing support of FreeBSD, 
the FreeBSD Foundation, and all those engaged in this 
amazing Project, which recently celebrated its 30th 
anniversary! Now, sit back, relax, and enjoy all the great 
content in this issue! 


Æ jfreeBso 


Deb Goodkin 
Executive Director of the FreeBSD Foundation 
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BY COLIN PERCIVAL 


large amount of fantastic open source software originates from “scratching an itch”. 
Such was the case with Firecracker: In 204, Amazon launched AWS Lambda as a 
‘serverless” compute platform: Users can provide a function — say, ten lines of Py- 
thon code — and Lambda provides all of the infrastructure between an HTTP request arriv- 
ing and the function being invoked to process the request and generate the response. 
To provide this service eff ciently and securely, Amazon needed to be able to launch vir- 
tual machines with minimal overhead. Thus was born Firecracker: A Virtual Machine Monitor 
which works with Linux KVM to create and manage “microVMs” with minimal overhead. 


In June 2022, | started work on porting FreeBSD to run 
on Firecracker. My interest was driven by a few factors. 
First, | had been doing a lot of work on speeding up 
the FreeBSD boot process and wanted to knowthe lim- PO rting FreeBSD to 
its that could be reached with a minimal hypervisor. 
Second, porting FreeBSD to new platforms always New platfo rms always 
helps to reveal bugs — both in FreeBSD and on those hel 
platforms. elps to reveal bugs — 
Third, AWS Lambda only supports Linux at present; 
I'm always eager to make FreeBSD more available in both in FreeBSD 
AWS (although adoption in Lambda is out of my control, and on those ol atforms. 
Firecracker support would be a necessary precondition). 
The largest reason, however, was simply because it’s 
there. Firecracker is an interesting platform, and | want- 
ed to see if | could make it work. 


While Firecracker was designed for Lambda’s needs — launching Linux kernels — there 
eh patches available from 2020 that added support for the PVH boot mode in addition 
o “linuxboot”. FreeBSD had support for PVH booting under Xen, so | decided to see if that 
ai work. 
Here | ran into the f rst problem: Firecracker could load the FreeBSD kernel into memory 
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but couldn't f nd the address at which to start running the kernel (the “kernel entry point”). 
According to the PVH boot protocol, this value is specif ed in an ELF Note — a piece of spe- 
cial metadata stored in ELF (Executable and Linker Format) f les. It turned out that there are 
two types of ELF Notes: PT NOTEs and SHT_NOTEs, and FreeBSD wasn't providing the 
one Firecracker was looking for. A small change to the FreeBSD kernel linker script f xed this, 
and now Firecracker was able to start executing the FreeBSD kernel. 
That lasted for about Lmicrosecond. 


Early Debugging 

FreeBSD has wonderful debugging functionality, but if your kernel crashes before the de- 
bugger is initialized or the serial console is set up, you're not going to get much help. In this 
case, the Firecracker process exited, telling me that the FreeBSD guest hit a triple-fault — 
but that’s all | knew. 

It turned out, however, that this was enough information to get me started, given a bit 
of creativity. If the FreeBSD kernel execution reached ah1t instruction, the Firecracker pro- 
cess would keep running but use 0% of the host’s CPU time (since it was virtualizing a halt- 
ed CPU). As such, | could distinguish between “FreeBSD is crashing before this point” and 
“FreeBSD is crashing after this point” by inserting a h1t instruction — if Firecracker exited, | 
knew that it was crashing before reaching that instruction. Thus started a process | referred 
to as “kernel bisection” — rather than bisecting a list of commits to f nd one which intro- 
duced a bug (asin git bisect)! would do a binary search through the kernel startup code 
to f nd the line of code that was making FreeBSD crash. 


Xen Hypercalls 

The f rst thing | discovered in this process was Xen FreeBSD has wonderful 
hypercalls. The PVH boot mode originated as the Xen/ l l 
PVH boot mode, and FreeBSD’s PVH entry point was, debugging functionality, 
in fact, an entry point specif cally for booting under Xen , 
— and the code made the quite reasonable assumption but if your kernel crashes 
that it was, indeed, running inside Xen and could thus bef he deb 
make Xen hypercalls. KVM (which provides the kernel efore the de ugger 
functionality used by Firecracker) is not Xen, of course, so initial; i 
so it doesn’t provide those hypercalls; attempting to use is initialized or the serial 
any of them resulted in the virtual machine crashing.AS Console is set up, 
an initial workaround, | simply commented out all the 
Xen hypercalls; later, | added code to check CPUID fora you're not going to get 
Xen signature before making calls e.g., to write debug- 
ging output to the Xen debug console. much help. 

There was one Xen hypercall that provided essential 
functionality, however: Retrieving the physical memo- 
ry map. (Of course, inside a hypervisor, the “physical” 
memory is only virtually physical. It’s turtles all the way down.) Here, we're saved by the fact 
that Xen/PVH was retroactively declared to be version O of PVH boot mode: From version 7 
onwards, a pointer to the memory map is passed via the PVH start_info page (a pointer 
to which is provided in a register when the virtual CPU starts executing). | had to write code 
to make use of the PVH version 7 memory map instead of relying on a Xen hypercall to get 
the same information, but that was easy enough. 

Another related issue arose from how Xen and Firecracker arrange structures in mem- 
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ory: Whereas Xen loads the kernel f rst and then places the start_info page at the end, 
Firecracker placed the start_info page at af xed low address and then loaded the kernel 
afterwards. This would have been f ne but that FreeBSD’s PVH code - having been written 

with Xen in mind — assumed that the memory immediately after the start_info page 

would be free for use as scratch space. Under Firecracker, that very quickly meant overwrit- 

ing the initial kernel stack — not an optimal outcome! A change to FreeBSD’s PVH code 

to assign scratch space after a// the memory regions initialized by the hypervisor f xed this 
problem. 


ACPI — or Lack Thereof! 

On x86 platforms, FreeBSD normally makes use of ACPI to learn about (and in some cas- 
es control) the hardware on which it is running. In addition to discovering things via ACPI 
which we might commonly think of as “devices” — disks, network adapters, etc. — FreeBSD 
also learns about fundamental things like CPUs and interrupt controllers via ACPI. 

Firecracker, being deliberately minimalist, does not implement ACPI, and FreeBSD gets 
upset when it can’t f gure out how many CPUs it has or where to f nd their interrupt con- 
trollers. 

Fortunately, FreeBSD has support for the historic Intel MultiProcessor Specif cation, 
which provides this critical information via an “MPTable” structure; it’s not part of the 
GENERIC kernel conf guration, but for running in Firecracker, we want to use a stripped- 
down kernel conf guration anyway, so it was easy 
to add device mptable to make use of what Fire- You can add options 
cracker provides. 

Except...it didn’t work. FreeBSD still couldn’t MPTABLE LINUX BUG 
f nd the information it needed! It turned out that 7 a E 
Linux has bugs in how it f nds and parses the MPT- COMPAT to your kernel 
able structure — and Firecracker, being designed . l 
to boot Linux, provided the M PTable in a way that conf guration if you need 
Linux supported but was not in fact compliant with TRET 
the standard. FreeBSD, having an implementation bug-for-bug compatibility 
independently written to follow the standard, failed with Linux’s MPTable 
both to f nd the (incorrectly located) MPTable and 
to parse the (invalid) MPTable once it was found. handli ng. 

So now FreeBSD has a new kernel option: You 
can add options MPTABLE_LINUX_BUG_COMPAT 
to your kernel conf guration if you need bug-for-bug compatibility with Linux's MPTable 
handling — and with that, FreeBSD managed to boot a bit further in Firecracker. 


Serial Console 

One of the few emulated devices — as opposed to virtualized devices like the Virtio block 
and network devices — provided by Firecracker is the serial port. In fact, in a common con- 
f guration, when you launch Firecracker, the standard input and output of the Firecracker 
process become the serial port input and output of the VM, making it seem like the guest 
OS is just another process running inside your shell (which, in a certain sense, it is). At least, 
that’s how it’s supposed to work. 

By this point in the process of bringing up FreeBSD inside Firecracker | was able to boot 
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a FreeBSD kernel with a root disk compiled into the kernel image — I didn’t have the virtual- 
ized disk driver working yet — and read all the console output from the kernel. After all the 
kernel console output, however, FreeBSD entered the userland portion of the boot process, 
and | got 6 characters of console output — and then it stopped. 

Funnily enough, I’d seen that exact symptom over ten years earlier, when | was f rst get- 
ting FreeBSD working on EC2 instances. A bug in QEMU resulted in the UART not send- 
ing an interrupt when the transmit FIFO emptied; FreeBSD wrote 4 bytes to the UART and 
then wouldn't write anymore because it was waiting for an interrupt which never arrived. 
Modern EC2 instances run on Amazon’s “Nitro” platform, but in the early days they used 
Xen and devices were emulated using code from QEMU. Somehow, a decade after this bug 
was f xed in QEMU, exactly the same bug was implemented in Firecracker; but luckily for 
me, the workaround | put into the FreeBSD kernel — hw. broken_txfifo="1" — was still 
available, and adding that loader tunable (which, since Firecracker loads the kernel directly 
without going through the boot loader, meant compiling the value into the kernel as an en- 
vironment variable) f xed the console output. 

| then found that the console input was also broken: FreeBSD didn’t respond to anything 
| typed into the console. In fact, tracing the Firecracker process, | found that Firecracker 
wasn’t even reading from the console — because Firecracker thought that the receive FIFO 
on the emulated UART was full. This turned out to be another bug in Firecracker: While ini- 
tializing the UART, FreeBSD f Ils the receive FIFO with garbage to measure its size and then 
f ushes the FIFO by writing to the FIFO Control Register. Firecracker didn’t implement the 
FIFO Control Register, so it was left with a full FIFO and quite sensibly didn’t try to read any 
more characters to put into it. Here, | added another workaround to FreeBSD: If LSR_RXRDY 
is still asserted after we try to f ush the FIFO via the FIFO Control Register — which is to 
Say, if the FIFO didn’t empty as requested — we now proceed to read and discard char- 
acters one by one until the FIFO empties. With this, Firecracker now recognized that 
FreeBSD was ready to read more input from the serial port, and | had a working bidirec- 
tional serial console. 


Virtio Devices 

While a system without disks or network could be useful for some purposes, before we 
can do very much with FreeBSD, we're going to want those devices. Firecracker supports 
Virtio block and network devices and exposes them to virtual machines as mmio (memo- 
ry-mapped I/O) devices. First step to getting these working in FreeBSD: Add device 
virtio_mmio to the Firecracker kernel conf guration. 

Next up, we need to tell FreeBSD how to f nd the virtualized devices. FreeBSD expected 
mmio devices to be discovered via FDT (Flattened Device Tree), which is a mechanism com- 
monly used on embedded systems; but Firecracker passes device parameters via the kernel 
command line with directives such as virtio_mmio.device=4K@0x1001e000:5. Second 
step to getting these working in FreeBSD: Write code for parsing such directives and creat- 
ing virtio_mmio device nodes. (Once we create the device node, FreeBSD’s regular pro- 
cess for device probing kicks in and the kernel will automatically determine the type of Vir- 
tio device and hook up the appropriate driver.) 

If we have multiple devices however — Say, a disk device and a network device — another 
problem arises: Firecracker passes directives the way Linux expects — as a series of key=val- 
ue pairs on the kernel command line — while FreeBSD parses the kernel command line as 
environment variables... meaning that if there were two virtio_mmio.device= directives 
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passed on the command line, only one was retained. To f xthis, | rewrote the early kernel 
environment parsing code to handle duplicate variables by appending a numbered suff x: 
We end up with virtio_mmio.device= for one device and virtio_mmio.device_1= for 
the second device. 

With this, | f nally had FreeBSD booting and discovering all of its devices — but one more 
problem arose with disk devices: If | shut down the virtual machine uncleanly, on the next 
boot the system would run fsck on the f lesystem, and the kernel would panic. It turned 
out that fsck is one of very few things in FreeBSD that will cause non-page-aligned disk I/Os, 
and FreeBSD’s Virtio block driver was causing a kernel panic when trying to pass unaligned 
I/Os to Firecracker. 

When an I/O crosses a page boundary — as will 
happen with page-sized I/Os which aren't aligned 
to page boundaries — the physical I/O segments 
will typically not be contiguous; most devices can 
handle I/O requests which specify a series of seg- Once] had FreeBSD 
ments of memory to be accessed. Firecracker, be- 
ing extremely minimalist, does not do this: Instead, Up and running in 
it accepts only a single data buffer — meaning that _. 

a buffer that crosses a page boundary can'tsim- Firecracker, it rapidly 

ply be split into pieces the way it would with other 

Virtio implementations. Fortunately, FreeBSD has became clear that there 

a system in place specif cally for taking care of de- 

vice complications like this:busdma. were some Improvements 
This was probably the hardest part of getting to be made. 

FreeBSD running in Firecracker, but after sever- 

al attempts, | think | f nally got it right: 1 modif ed 

FreeBSD's Virtio block driver to use busdma, and 

now unaligned requests are “bounced” (aka. cop- 

ied via a temporary buffer) in order to comply with 

the limitations of the Firecracker Virtio implementation. 


Revealed O ptimizations 

Once | had FreeBSD up and running in Firecracker, it rapidly became clear that there 
were some improvements to be made. One of the f rst things | noticed was that, despite 
having 28 MB of RAM in the virtual machine | was testing, the system was barely usable, 
with processes being frequently killed due to the system running out of memory. The 
top(1) utility showed that almost half of system memory was in the “wired” state, which 
seemed odd to me; so | investigated further, and found that busdma had reserved 32 MB of 
memory for bounce pages. This was clearly far more than needed — given Firecracker’s lim- 
itations and the fact that bounce pages are generally not allocated contiguously, each disk 
I/O should use at most a single 4 kB bounce page — and | was able to reduce this memory 
consumption to 52 kB with a patch to busdma which limited its bounce page reservations 
for devices which supported only a small number of I/O segments. 

Once the system was more stable, | started paying attention to the boot process. If 
you're watching a system boot and there’s suddenly a pause in the messages scrolling past, 
there's probably something happening at that point which is slowing down the boot pro- 
cess. Simple eyeballing of the boot process — and also the shutdown process — revealed 
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several improvements: 

e FreeBSD’s kernel random number generator usually obtains entropy from hardware de- 
vices, but in virtual machines this may not be an effective source. As a backup source of 
entropy, on x86 systems we make use of the RDRAND instruction to obtain random val- 
ues from the CPU; but we were only obtaining a very small amount of entropy on each 
request and were only requesting entropy once every 100 ms. Changing the entropy 
gathering system to request enough entropy to fully seed the Fortuna random number 
generator shaved 2.3 seconds off the boot time. 

When FreeBSD f rst boots, it records a Host ID for the system. This is typically obtained 
from hardware via the smbios.system.uuid environment variable, which the boot 
loader sets based on information from BIOS or 
UEFI. Under Firecracker, however, there is no 
boot loader — and thus no ID being provided. 

We had a fallback system in place that would 

generate a random ID in software on sys- 

tems that didn’t have a valid hardware ID; but 

we also printed a warning and waited 2 sec- IPv6 mandates that 
onds to allow the user to read it. | changed this 

code to print the warning and wait 2seconds systems wait for 

if the hardware provided an invalid ID, but pro- 

ceed silently and quickly if the hardware simply “Dupl icate Address 
didn't provide an ID. — i 

*IPV6 mandates that systems wait for “Dupli- Detection” before using 
cate Address Detection” before using an IPv6 
address. In rc.d/netif, after bringing up in- an IPv6 address. 
terfaces, we were waiting for IPv6 DAD if any 
of our network interfaces had IPv6 enabled. 

There’s just one problem with that: We always 

have IPv6 enabled on the loopback interface! 

| changed the logic to only waiting for DAD 

if we had IPv6 enabled on an interface other 

than the loopback interface and sped up the boot process by 2 seconds — if another 
system is trying to use the same IPv6 address as us On our 100, we have bigger prob- 
lems than an address collision! 

eWhen rebooting, FreeBSD printed a message (‘Rebooting...”) and then waited 1sec- 
ond “for printf’s to complete and be read”. This seemed minimally useful, since people 
can usually tell that the system is rebooting — there isnowakern.reboot_wait_time 
sysctl which defaults to Zero. 

eWhen shutting down or rebooting, the FreeBSD BSP (CPU #0) waits for the other CPUs 
to signal that they have stopped. ..and then waited an extra lsecond to make sure that 
they had a chance to stop. | removed the extra second of wait time. 

Once the low-hanging fruit was out of the way, | pulled out TSLOG and started looking 
at f amecharts of the boot process. Firecracker is a great environment for doing this, for two 
reasons: First, the minimalist environment eliminates a lot of noise, making it much easier 
to see what’s left behind; and second, having Firecracker launch virtual machines extremely 
quickly made it possible to test changes to the FreeBSD kernel very rapidly — often well un- 
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der 30 seconds to build a new kernel, launch it, and generate a new f amechart. 

Investigation with TSLOG revealed a number of available optimizations: 
elapic_init had a 00000-iteration loop to calibrate how long lapic_read_icr_lo 
took to execute; cutting this down to 1000 iterations shaved off 10 ms. 

ens8250_drain called DELAY after each character was read; changing this to check for 
LSR_RXRDY and only DELAYing if nothing was already available to be read shaved off 27 

ms. 

e FreeBSD makes use of a CPUID leaf which most hyperuisors use to advertise the TSC 
and local APIC clock frequencies; Firecracker, unlike VMWare, QEMU, and EC2, did not 
implement this. Adding support for this CPUID leaf to Firecracker shaved 20 ms off the 
FreeBSD boot time. 

e FreeBSD was setting kern.nswbuf (which controls the number of buffers allocated for 
a variety of temporary purposes) to 256 regardless of the size of the system; changing 
this to 32 * mp_ncpus shaved 5 ms off the boot time on a small (LCPU) virtual ma- 
chine. 

¢FreeBSD's mi_startup function, which kicks off machine-independent system initial- 
ization routines, was using a bubblesort to order the functions it called; while this was 
reasonable in the 90s given the small number of routines needing to be ordered at that 
point, there are now over 1000 such routines and the bubblesort was getting slow. Re- 
placing it with a quicksort will save 2 ms. (Not yet committed at press time.) 

¢FreeBSD's vm_men initialization routine was initializing vm_page structures for all avail- 
able physical memory. Even on a relatively small VM with 28 MB of RAM, this meant 
initializing 32768 such structures — and took a few ms. Changing this code to initialize 
vm_page structures “lazily” as the memory is allocated for use will save 2 ms. (Not yet 
committed at press time.) 

e Firecracker was allocating VM guest memory via an anonymous mmap, but Linux was 
not setting up the paging structures for the entire VM guest address space. As a result, 

the f rst time any page was read, a fault would occur taking roughly 20,000 CPU cycles 
to be resolved while Linux mapped in a page of memory. Adding the MAP_POPULATE 
f ag to Firecracker’s mmap call will save 2 ms. (Not yet committed at press time.) 


Current Status 
FreeBSD boots under Firecracker — and does so extremely quickly. Including uncommit- 
ted patches (to FreeBSD and also to Firecracker), on a virtual machine with LCPU and 28 
MB of RAM, the FreeBSD kernel can boot in under 20 ms; af ame chart of the boot process 
appears below. 
14.0-CURRENT boot 
19496 us 


| 


There is still work to be done: In addition to committing the patches mentioned above 
and getting PVH boot mode support merged to “mainline” Firecracker, there’s a signif - 
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cant amount of “cleanup” work to be done. Due to the history of PVH boot mode originat- 
ing from Xen, the code used for PVH booting is still mixed up with Xen support; separating 
those will simplify things signif cantly. Similarly, it’s currently impossible to build a FreeBSD 
arm64 kernel without PCI or ACPI support; f nding the bogus dependencies and remov- 
ing them will allow for a smaller FreeBSD/Firecracker kernel (and also shave off a few more 
microseconds from the boot time - we spend 25 us checking to see if we need to reserve 
memory for Intel GPUs). 

More aspirationally, it would be great to see if Firecracker could be ported to run on 
FreeBSD — at acertain point, a virtual machine is a virtual machine, and while Firecracker 
was written to use Linux KVM, there’s no fundamental reason why it shouldn't be possible 
to make it use the kernel portion of FreeBSD’s bhyve hypervisor instead. 

Anyone wanting to experiment with FreeBSD in Firecracker can build a FreeBSD 4.0 
kernel with the amd64 FIRECRACKER kernel conf guration, and check out the feature/pvh 
branch from the Firecracker project; or if that branch no longer exists it means the code has 
been merged into the mainline Firecracker tree. 

If you try out FreeBSD on Firecracker — especially if you end up using it in production — 
please let me know! | started this project mainly out of interest, but I'd love to hear if it ends 
up being useful. 


COLIN PERCIVAL has been a FreeBSD developer since 2004 and was the project’s 
Security Off cer from 2005 to 202. In 2006, he founded the Tarsnap online backup 
service, which he continues to run. In 2019, in recognition of his work bringing FreeBSD 
to EC2, he was named an Amazon Web Services Hero. 


Contact Jim Maurer 
with your article ideas. 


(maurer.jim@gmail.com) 
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BY LUCA PIZZAMIGUIO._“ 


servers. When the number of applications and their cardinality grows, the number of 
= containers can easily become hard to manage manually. 
Container orchestrators are applications that aim to simplify the management of a large 
f eet of containers, hiding the complexity and improving reliability, especially in an environ- 
ment made very dynamic by autoscaling and continuous deployment. In this article, we will 
talk about a setup based on FreeBSD, using pot, a jail framework that supports jail images, 
and nomad, a container agnostic orchestrator developed by HashiCorp. 


C ontainers are a great tool to distribute horizontally scalable applications on many 


The System Architecture 

To explain how an orchestrator works, we need to introduce a few services and explain 
their role. 

The nomad client 

A nomad client is a server that receives orders from the orchestrator to execute contain- 
ers. Nomad clients can also be referred to as nodes, like in kubernetes. 

In large installations, the majority of servers are nomad clients, as they are responsible to 
execute users’ applications. In the cloud native jargon, nomad clients form the data plane of 
the cluster. 

Nomad clients can support multiple container drivers: some drivers are operating sys- 
tem agnostic, while other drivers, like docker or pot, are available only on specif c operat- 
ing systems. 

To orchestrate FreeBSD jails using pot, we need nomad clients based on FreeBSD. 
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The nomad server 

The nomad servers are the machines that implement the orchestrator. The nomad serv- 
er is responsible for keeping the state of the cluster and for scheduling containers to nomad 
clients. Multiple instances (3 to 5) are required to provide redundancy and to share the load. 
In the cloud native jargon, nomad servers form the control plane of the cluster. 

Users are going to interact with nomad servers to deploy applications to the cluster. The 
nomad servers are responsible to keep the state of the cluster healthy, and to reschedule 
containers in case of client failures. 

Nomad servers can run on any supported operating system. To orchestrate jails, at least 
one nomad client has to be based on FreeBSD. 

The container registry 

The orchestrator (nomad servers) is the “brain” of the cluster and it assigns containers to 
clients that support the container types. This means that: 

any nomad clients can be selected to run supported containers, 

eclients don’t know in advance which containers are going to host, 

ea client can execute multiple instances of the same containers. 

Every client needs a service to download the image of the container they are assigned 
to execute. This requirement is fulf Iled by the container registry, a service that provides the 
images of containers. 

When the orchestrator selects a client to execute a container, that client is going to 
download the container image from a registry and then start the container. 

In our examples, we are going to use potluck, a pub- 
lic container registry maintained by the pot community, 
that builds images from an open source catalog of im- 
age recipes. However, as container images contain bina- 
ries, we strongly encourage everyone to have their own 
local registry, for security reasons. 

In pot, images are f les that are downloaded via Users are going 
fetch(1), so a registry can be just a simple web server. to interact with nomad 
The service catalog 

A service catalog is a list of services enriched byad- servers to deploy 
ditional information like all the container addresses that a 
implement those services. applications 

When an orchestrator schedules a container imple- 
menting a service, it’s also going to register the contain- tO the cluster. 
er address to the list of containers implementing that 
service. 

The service catalog can be also conf gured to period- 
ically check the health status of the service for all con- 
tainers, so the list of container addresses only contains 
healthy addresses. 

The service catalog that we are going to use is based on consul, a service mesh applica- 
tion also developed by HashiCorp. 

The ingress (optional) 

Because of the dynamic nature of the orchestrator, it can be very hard to know where 

our services are running. Every time a new deployment occurs, containers are scheduled to 
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potentially different nodes on different ports. With ingress, we def ne a proxy/load balancer 
that is conf gured to provide af xed entry point to our services. 

For instance, we can conf gure the proxy in a way that the URL path (i.e., https://example. 
com/foo) provides information about the targeted service (i.e. redirecting to the containers 
implementing the service foo) Another common way is to use the host header. 

Ingress proxies dynamically maintain the list of valid container’s addresses, by continuous- 
ly interacting with the service catalog. 

For our examples, we are using traef k, an ingress proxy implemented by traef x lab. 


Nomad-pot-dnver 

Nomad has been built to support several container technologies and different operating 
systems. In fact, a nomad package Is available and HashiCorp also provides binary blobs for 
FreeBSD. 

Nomad has a plugin architecture that allows it to extend it to support new container 
technologies. Esteban Barrios wrote and opensourced the nomad-pot- driver plugin. This 
plugin works as an interface between a nomad client and pot, to provide the features need- 
ed to orchestrate jails. 

The orchestrator schedules workloads to the client that uses the plugin to interface with pot. 


Control flow nomad client 


Data flow nomad 


driver 
a Job start agi 


Developer Users 
Overview of the architecture and the foles of the different services. 


Minipot 

Minipot is a package that installs and conf gures all the aforementioned services on a sin- 
gle FreeBSD machine and is the reference installation that we are going to use to show our 
examples. 

Minipot is a conf guration useful for testing, but not for professional installation, as it’s 
going to concentrate all service on one machine only, reducing a cluster to a single node 
that does it all. 

In particular, it is going to install and conf gure consul, traef k, and nomad. Nomad is go- 
ing to run as client and as server, playing the dual role of orchestrator and executor. 

There is a detailed guide on how to install minipot on this blog post on the Klara website. 


Schedule a Job 
Once minipot is initialized and all services are running, we can use the following job de- 
scription f le to start a job in nomad 
1 job "nginx-minipot" { 
2 datacenters = ["minipot"] 
3 type "service" 
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group "groupi" { 
count = 1 
network { 
port "http" {} 
li 


task "wwwi" { 
driver = "pot" 


service { 
tags = ["nginx", "www"] 
name = "hello-web" 
port = "http" 


check { 
type = "tcp" 
name = "tcp" 
interval = "5s" 
timeout = "2s" 
} 
} 


config { 
image = "https://potluck.honeyguide.net/nginx-nomad" 
pot = "nginx-nomad-amd64-13_1" 
tag = "1.1.13" 
command = "nginx" 
args = ["-g","’daemon off;’"] 


port_map = { 
http = "80" 
} 
F 


resources { 
cpu = 200 
memory = 64 
} 
} 
} 
} 


The job stanza describes all the details nomad needs to schedule the job. 

The job “nginx minipot” (1) has one group named “group? (4), that has one task called 
“wwwt' (9). 

The task “www? is a pot container (10), the image registry is potluck (23), the pot image is 
nginx (24) and the version is 113 (25). By specifying that the task is based on the pot driver, 
the orchestrator is going to schedule this job on a client with pot support. In our example, 
server and client are the same machine. 

Compared to the common use of jails, where the bootstrap happens using rc script, we 
are going to run nginx directly (26), without any other additional services. The args parame- 
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ter (27) is important to allow nomad to properly follow the container lifecycle and to capture 
its logs. Pot is going to take care of initializing the network and everything that is needed. 

The port_map stanza (28) and the network stanza (6) are saying that nginx is going to lis- 
ten to port 80 in the jail, but the nomad client will use a different port (“http”), dynamically 
assigned by the nomad server. 

The service stanza (1) provides the information that nomad is going to use to register 
the service to consul. In our example, the task “wwwI’ implements the service “hello-web” 
(1) and it will be registered to consul, using the port “http” (74), assigned by the nomad serv- 
er. For the IP address, the nomad server is going to use the nomad client IP address, def ned 
during the scheduling operation. 

In our example, the job is also conf guring a tcp health check that consul is going to run 
every 5 seconds to determine the health status of the instance. 

This job description needs to be stored in af le (i.e., nginxjob) and any user can launch 
the service via 


$ nomad job nginx.job 


NOTE: The f rst deployment can take some time as the client needs to download the im- 
age. On slow connections, the f rst deployment can even fail because of deployment time- 
outs. Once the fetch is completed, it is safe to re-run the deployment that will be executed 
in few seconds. 

Check on nomad 

Once the job is scheduled, it is possible to check the status of the deployment via com- 

mand line: 


$ nomad jobs allocs nginx-minipot 

ID Node ID Task Group Version Desired Status Created Modified 
636d3241 c375b833 groupl 3 run running 28m43s ago 28m27s ago 
$ nomad alloc status 636d3241 

E 

Allocation Addresses: 

Label Dynamic Address 

*http yes 2003:f1:c709:de00:faac:65ff : fe86 : 9458 : 22854 

fs sad 

$ curl "[2003:£1:c709:de00:faac: 65ff : fe86:9458] : 22854" 


“Allocation” is the name used by nomad to identify a container, an instance of the task. 

The IPv6 address is the nomad client one. 

The port 22854 is the port nomad chose to direct the nomad client to the port 80 of the 
container. 

To see the port redirection setup, we can use the command: 


$ sudo pot show 


An alternative of the CLI, the nomad server is conf gured to also provide a powerful web 
Ul reachable at localhost:4646 

From there, we can see the “nginx minipot” job and navigate to see all information about 
the cluster, allocations, clients, and so on. 

From nomad, we can see directly the status of all containers. 

By reaching the allocation page, we can click on the “exec” button to run a/bin/sh shell 
into the running containers. 
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Check on consul 

Via CLI, we can see the list of services in the consul catalog (consul catalog services), but 
not the details. However, we can check the status of the “hello-web” service, by reaching the 
web UI interface at 


localhost :8500 


From here, we can navigate to check the status of the service “hello-web” and its check 
“tcp”. 
Check on traef k 

The proxy traef x is conf gured to route traff c on port 8080, but provides a web-ui to 
monitor the status at port 9200 (localhost:9200). Traef k is also conf gured to sync the 
service catalog from consul. 

By selecting the http services, we can see our “hello-web” service (marked as 
hello-web@consulcatalog). 

By clicking on the service, we can see the service details and the routing options. 

The conf guration is based on the host header, in our case “hello-web.minipot”. 

Now we can reach the service hello-web via the ingress: 


$ curl -H Host:hello-web.minipot http://127.0.0.1:8080 


Alternatively, we can add the entry 


127.0.0.1 hello-web.minipot 


to /etc/hosts and then use the hostname directly: 


$ curl http://hello-web.minipot :8080 


We will obtain the same output as we had by point curl directly to the jail. 
Horizontal scaling 

To see the orchestrator in action, we can now simply change the count from 1to 2 in the 
job f le (row 5) and re-submit the job via 


$ nomad run nginx.job 


Once scheduled, the running nomad allocations are now 2, the service “hello-web” in 
consul has two instances, as the servers in traef k. 

e To verify the round-robin distribution of the ingress, we can 

e Tail the logs of one container ($ nomad alloc logs -f allocation1) 

e Tail the logs of the other container ($ nomad alloc logs -f allocation2) 

eExecute curl on the ingress ($ curl -H Host:hello-web.minipot 

http://127.0.0.1:8080) 

At every execution of curl, the proxy distributes the requests between the containers as is 
possible to see from their logs. 
Tear down everything 

To stop our examples, we suggest this tear down operation: 

eStop the nomad jobs ($ nomad stop nginx-minipot) 

eStop traefk($ sudo service traefik stop) 

eStop nomad ($ sudo service nomad stop) 

eStop consul ($ sudo service consul stop) 
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From playground to production 
Minipot is a single node installation useful as a playground, to learn or to test things locally. 
A production environment should be shaped in a different way: 
¢3 or 5 distinct consul servers 
¢3 or 5 distinct nomad servers 
e2 ingress proxy servers (HA conf guration) 
Several nomad client servers (depends on the expected workload and reliability require- 
ments like the overprouision factor) 
It is worth mentioning that the aforementioned set- 
up can mix different operating systems: the only servers 
that have to run FreeBSD are the nomad clients target- 
ing jail/pot workloads. 
Nomad or consul servers can run on Linux or Solaris, 
allowing you to reuse pieces of infrastructure that may TEF ; 
already be available to you. M Inipot ISa single node 


As ingress proxy, we used traef k, which natively syncs ; . 
with consul. However, it’s possible to use other services, installation useful 


like nginx or ha-proxy, together with consul-template, asa ol ayg round 

to achieve the same result. In such a conf guration, con- f 

sul-template is responsible for detecting changes in to learn or to 

consul,to render the proxy conf guration template, and 

to notify the proxy of the new conf guration. test things locally. 
Additionally, the conf guration of all services would 

need improvements, like adding authentication to 

nomad. 


Credits 

| want to highlight the community effort needed 
to reach this point, from Esteban Barrios, the f rst developer of the nomad-pot-driver, to 
Michael Gmelin (grembo@), who really helped a lot to increase the reliability and the sta- 
bility of this solutions. | also want to mention Stephan Lichtenauer and Bretton Vine, who 
worked on Potluck, the public image registry, and on numerous other projects like the An- 
sible playbook and blog posts dedicated to pot and nomad use cases. 
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derstanding how containers on FreeBSD could look. 
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-A Theologians Report of His Trip to Present aBSDC an 2023 
BY COREY STEPHAN 


S| sit before my battle-worn Think- 
Pad running OpenBSD 73-current on 
the Air Canada f ight from Toronto, 
Ontario to Houston, Texas to return 
home to my wife, children, and (ir)regular job as 
Assistant Professor of Theology and Fellow of the 
Core at the University of St. Thomas, after my 

f rst experience at any conference about comput- 
er science or information technology, | feel tired 
but content. | departed the conference wearing 
my mustard OpenBSD 72-release t-shirt with its 
Dr. Seuss theme: “One diff, two OKs, commit, blowf sh” (a parody of “One f sh, two f sh, red 
f sh, blue f sh”). Realizing that | am once again in the real world in which normal people, sad- 
ly, do not (yet?) run BSD operating systems on their laptop computers, | chuckle to myself 
while musing about the (distantly remote) possibility that my private security screening back 
at the Ottawa International Airport was not a mere coincidence. After all, this t-shirt rep- 
resents an operating system whose founder, Theo de Raadt, famously relocated the project 
from the United States to Canada due to American laws about the export of cryptography. 

| jokingly wonder if this might be the f rst time someone 
has mistaken me for a hacker (of the malignant variety). 
If so, have | just endured another rite of passage into the 
BSD crowd? 

Asan obvious outsider to BSDCan who made a point 
of introducing myself to many fellow attendees, | re- 
a ceived myriad questions. Oddly, the most common was 

4 oe the most challenging for me to answer: Corey, why are 
* you here? BSDCan regulars were genuinely curious as 
to what a professional, Catholic theologian was doing at 
a conference about Unixlike operating systems. Even 
now, after the conference has ended, | am not sure why 
| journeyed to Ottawa for BSDCan 2023. Above all, | sup- 
pose that | wanted to try something wholly different 
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from my day-to-day work. For years, | have been an amateur learning about the BSDs from 
Michael W. Lucas’s Absolute FreeBSD (3rd edition) and Absolute OpenBSD (2nd edition), as 
well as the online recordings of talks from BSDCan, EuroBSDCon, and AsiaBSDCon (and the 
Youtube personality RoboNuggie). Although | occasionally engage in simple homelabbing, 
my main interest in FreeBSD, OpenBSD, and the other BSD operating systems long has 
been their common utility as fully customizable, desktop operating systems. Specif cally, | 
like to use FreeBSD and OpenBSD to aid my multisource research and writing as a scholar of 
the history of Christian theology. My October 2021FreeBSD es lecture “FreeBSD for the 
Writing Scholar” was about that very theme, as was y i 
my talk at BSDCan, “BSD for Researching, Writing, 
and Teaching in the Liberal Arts.” | 
At a standard academic conference, dressing the i 
part of the ref ned scholar is important, especially Indeed, if creativi 
in our anti-intellectual, twenty-f rst century West- : 
em context in which, say, one English literature job is found at the 


posting attracting hundreds of Ph.D-holding, qual- intersection of aoparentl 
if ed (and desperate) applicants has become the Pp y 


norm. While packing to present at BSDCan, | al- disconnected subjects, 
most instinctively grabbed my tailored navy-blue 

suit with my favorite golden bow tie. By Providence, then | am pleased to 

| had the wherewithal to write a short email to Dan 

Langille, one of the founders of BSDCan who co- report that | had several 
ordinated the conference for twenty years before 
announcing his well-earned retirement from that engaging discussions 


duty during this year’s closing session, to check how at BSDCan that were 
attendees and presenters typically dress. His reply 


was that he would be wearing cargo shorts and a themselves loci of creative 
t-shirt through the whole conference, and | recalled 

that | had watched several BSD(Can/Con) video ree genlus. 

cordings in which the speakers were dressed in kind 

(Theo de Raadt in shorts and sandals, Michael Lu- 

cas in t-shirts with cartoonish f gures of horror, and 

SO On). Recognizing that | would be laughably out 

of place if | were to continue with my f rst sartorial selection but being unable to bring my- 
self to deliver a formal presentation without a shirt and tie, | compromised by packing one 
set of my standard university (teaching and meeting) attire (in this case, purple pants and 

a purple bowtie with grey suade shoes — no jacket). Even with that change, Michael Lucas 
told me after my talk that | was “the best dressed ... speaker that we ever have had.” | also 
made sure to have my full collection of BSD shirts with me to wear for the rest of the con- 
ference, that is, my FreeBSD polo shirt that was a gift from the FreeBSD Foundation when 

| presented for FreeBSD Friday, the Seussical O penBSD t-shirt that | was wearing as | began 
drafting this report, and a few old-stock-but-new OpenBSD t-shirts that were given to me 
by a generous Unix greybeard who wishes to remain anonymous. 

After presenting at many academic conferences on the topics of theology and/or early 
to medieval Church history over the years, | had grown accustomed to a certain set of cul- 
tural and behavioral expectations about conferences that were in no way applicable to BSD- 
Can. Starting with the rather silly point of dress, BsDCan presented me with a seemingly 
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unending series of opportunities to think freshly. 
Indeed, if creativity is found at the intersection 
of apparently disconnected subjects, then | am 
pleased to report that | had several engaging dis- 
cussions at BSDCan that were themselves loci 
of creative genius. In chronological order, here 

m are three such chats: Michael Lucas of technical 
= authorial fame talked with me about a few pos- 
sibilities for an educational book project that | 
have had in mind for some time; Tom Jones, whose voice | recognized from the BSD.Now 
podcast that | often enjoy during my long commutes in Houston, suggested that | ought 

to write this report for the FreeBSD Journal (for which he sits on the editorial board); and Dr. 
Marshall Kirk McKusick, original BSD Unix contributor and (great-)grandfather of FreeBSD 
who still commits code, explained that the famous Beastie character is a Unix background 
daemon as imagined by a Disney artist. 

To McKusick’'s story about Beastie, the Unix background daemon, | replied that the over- 
all rationale behind the artistry makes sense. In ancient Greek, | noted, a Satuwv (daemon) 
refers to aroaming spirit who either helps or hin- 
ders humans invisibly (in the background, like a 
Unix daemon). Since a daemon might either be | 
benevolent or malevolent, but the Devil of the : 

Christian tradition (obviously) is only malevolent, a In ancient Greek 
daemon is not the same as the Devil. | remain un- | 
convinced that Beastie’s yellow pitchfork and red af (daemon) refers — 


horns, inevitably offensive to some, are necessary, toa roaming spirit who 
since ancient daemons rarely were depicted in 


artwork — and certainly never as such. That mat either helps or hinders 
ter, however, seems best left alone until the next ae 
time that | chat with the ever-smiling McKusick. humans invisibly 
Besides, | cannot blame a gifted Disney animator ? 
for lacking familiarity with Greco-Roman folklore, (in the background, 
and I rather enjoy the medieval character that i ‘ 
graces (or curses) FreeBSD communication zones like a Unidaemon). 
to this day. 

Speaking of McKusick, on my f rst night at 
BSDCan (Wednesday), | stumbled into the end- 
of-day hackathon portion of the FreeBSD Developer's Conference. | was impressed to ob- 
serve McKusick collaboratively coding with someone who must have been at least forty 
years younger than he. The spirit of working to keep the BSD operating system projects 
multigenerational ran through the entirety of BSDCan. The oldest, most accomplished par- 
ticipants, who might have had legitimate reasons to ignore neophytes, especially outsiders 
(as | was), treated me as a worthy discussion partner. During that evening session, a few of us 
who are husbands and fathers talked about our wives and children, with the discussion f ow- 
ing naturally between the serious and the amusing before returning to BSD. 

| am not ashamed to admit that | did not understand everything in the lectures | attend- 
ed. Understanding everything never was my objective, nor should it be the objective of any- 
one attending any conference, be it academic, technical, spiritual, or other. | went to BSD- 
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Can in search of a novel exchange of ideas, sharing what | know with persons from outside 
my normal circles and, more importantly, learning from what such persons know. Although 
| often was reduced to attempting to intuit requisite background knowledge in real-time, 

| enjoyed nearly all the talks | attended. | certainly learned something in every session. Bra- 
vo, BSDCan’s organizational team, for electing 

a superb cluster of speakers (if | may be allowed 

to write that as someone who was one of those 
speakers). 

Tom Jones's “Making FreeBSD QUIC” and 
Marshall Kirk McKusick’s “Gunion(8): a new 
GEOM utility in the FreeBSD Kernel” made Fri- 
day, the f rst day, a joy, since both Jones and McK- Bravo,BSDC ans 


usick know how to work an audience with what org anizational team for . 
seems to be magical charm. (How, | wonder, does 


Dr. McKusick make a history of f le system mi- electing a superb cluster 
nutia almost as intriguing as one of Ken Burns's 

documentaries about a great American war?) Ad- Of speakers. 

ditionally, recording the 52th episode (not 500th, 
but 522th, since 522 is the important number in 
computing) of BSD.Now before a packed audi- 
torium, was a charming idea that the podcasting 
crew ought to repeat in the future. Shockingly, 
(the same) Tom Jones proclaimed to the entire 
conference that he was most looking forward to 
my talk about BSD and the liberal arts. | have no 
doubt that his proclamation boosted my session's 
turnout. 

On Saturday, the second day, Philipp Buehler’s 
“‘Jitsi on OpenBSD - Puffy presents video confe- MEN 
encing” included a daring live demonstration of his ~> 4 
working OpenBSD-hosted Jitsi server in which sev- 
eral of my fellow audience members concurrently 
connected. For that alone, even before he deliv- 
ered the rest of the stellar talk, Buehler earned my f rm applause. The CEOs of leading-edge 
technological f rms as well known as Elon Musk and the late Steve Jobs have embarrassed 
themselves by attempting to give live demonstrations before large audiences only to have 
the technologies that they have intended to showcase fail in real-time. Buehler, however, did 
not strike me as a fool; rather, he was conf dent enough in his implementation to demon- 
strate it to the world. Brooks Davis's “Creating a memory-safe workstation with CheriBSD” 
was about the application of leading-edge technical research from Cambridge University, 
so understanding it would have required a huge amount of background knowledge that | 
lacked. Worse, Davis's talk immediately preceded my own. I observed that Davis is a f ne pre- 
senter with a jolly demeanor, but | must humbly admit that almost all his talk entered one of 
my ears and escaped the other while | anxiously awaited my slot. 

Finally, my time came. Tom Jones had primed the rest of the conference attendees for 
me the day before, so the decently sized, University of Ottawa classroom in which | had 
been assigned to present was bustling. All morning, | had observed the weary faces that are 
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typical of the second day of every conference that | have attended, perhaps made worse at 
BSDCan because the conference organizers encouraged folks to visit a pub for local brews 
and then sl awake until (if not past) midnight collaboratively hacking. While | am neither 

| a drinker nor a hacker, | appreciate that the bleary eyes that 
f lled the conference halls and rooms on the second day 
were those of men and women who passed the night in 
lighthearted comradery while building a better technolog- 
ical future for us all. Yet, perhaps because | was standing 
at the front of a university classroom, wearing my normal 
teaching attire, | immediately snapped into my workplace 
modus operandi as a whimsical Taan professor of the 
traditional liber- FENTER N 
al arts. The room 
lacked the vital- 
ity that | need- 
ed to succeed as an i oddball. Accordingly, | called 
upon everyone in the room to rise, greet her or l hope to return 


his neighbor, and say, “It is a blessing to be sitting to interact with everyone 
next to you today.” 


My use of a classic professorial trick seems to I met at BSDCan 2023 
have succeeded. The audience members’ reac- , 
tions to my talk were loud, energetic, and wonde- again at BSDCan 2024 — 
ful from start to f nish. One of the crowd's favorite . 
moments in my talk was when | pointed to there. and to present something 
cording camera in the back of the room and ad- ® 
dressed my colleagues with this original line: “C is even more daring. 
to BSD what Latin is to us.” 

The Question-and-Answer session for my talk 
went over time, with Dr. McKusick himself asking 
several questions. The hallway discussions after- 
ward were engaging. Overall, my talk’s warm reception elated me. 

Without an obvious way to conclude this report, | will write two things. First, if you are 
interested in the BSD operating systems but nervous about BSDCan, do not be. While the 
conference’s regulars form a self-selected, tightly knit group, they are genuinely welcom- 
ing, and it does not take long for newcomers to start to assimilate into that group. Further, 
there is no shortage of entertainment at BSDCan (the charity auction is hilarious), and the 
talks are scheduled so that no one part of the conference becomes painful. Second, | hope 
to return to interact with everyone | met at BSDCan 2023 again at BSDCan 2024 — and to 
present something even more daring. 


DR. COREY STEPHAN serves as Assistant Professor of Theology and Fellow of the Core 
at the University of St. Thomas in Houston, Texas. He proudly makes exclusive use of 
free and open source software tools, including *BSD operating systems, to assist both 
his research in Catholic historical theology and his teaching of the traditional liberal arts. 
His professional website is coreystephan.com. 


Photos courtesy of Tom Jones. 
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Recollections: 
An Interview with 
Doug Rabson (dfr@) 


BY TOM JONES 


f the project and the people behind our favorite Operating System haven't been cov- 
ered in much detail. As a part of FreeBSD’s 30th Anniversary, | set out to speak to those 
involved at the start of development. 
This installment is with Doug Rabson, who has been a FreeBSD committer since 1994 
and is currently working on improving FreeBSD support for modern container orchestra- 
tion systems such as podman and kubemetes. 


IE FreeBSD project started out with contributions from many hands, but the early days 
O 


TJ: Could you explain—generally—what you were up to in the late 980s and early 990s 
before the project? 


DR: | graduated from college in the early 980s. Before that, | had been exposed to BSD 
and we used it on some of the machines there, but | didn’t pursue that kind of operating 
systems focus at all. | went to work for a little games company that some friends had start- 
ed, and we told them that Unix is cool, that they 

should absolutely get Unix if they were going to 

buy a computer. They bought a MicroVAX, put 

Ultrix on it, which was 4.2 BSD—approximate- | | remember when the 4.5 BSD 
ly—and we wrote Magnetic Scrolls. We were tapes were released, | got a copy 
writing interactive f ction using Unix systems l | 
because the micros we were targeting were too Trom a friend at Imperial College, 
weak. l University of London. 

That background interest stayed with me. 
| remember when the 4.3 BSD tapes were re- 
leased, | got a copy from a friend at Imperial 
College, University of London. Just for curiosity, 
| wanted to understand how it worked. The idea of being able to read the source code was 
very cool. | read through it, f gured out some of the pieces that were missing—and f gured 
out that | didn’t know enough to try to f Il in the gaps. 

A bit later, and this is again still just before FreeBSD, | heard about the 386BSD precursor 
to the Open/Net/FreeBSD group. Somebody did f Il in the gaops—somebody who actually 
knew what they were doing. And I installed that on some scratch hardware | found at work. 
And it worked. It didn’t work very well. | mean, it was sort of broken. And a bunch of peo- 
ple like Jordan—and I've forgotten all the names—N athan Whitehorn, | think, David Green- 
man. Anyway, they got together with a set of patches for 386BSD, because the author 
wasn’t really interested in doing much more with it. At that point, he'd written his article for 
Dr. Dobbs and had done the work. But he wasn’t taking patches. There was a kind of or 
ganic movement to collate the patches to various bugs and features that had been added 
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to 386BSD. And that was the 386BSD patch kit, which went through a number of versions. 
When it was clear that the author, Mr. Jolitz, wasn’t really interested in taking his project fur- 
ther, that turned into the BSDs. 

That was the point where Net and Free split. FreeBSD people wanted to target one via- 
ble platform, which is the 386, PC-based commodity hardware. The NetBSD people were 
interested in holding on to the portability that was always there in the BSD platform. And 
so, they diverged at that point. | don’t think there were any hard feelings. It was just a dif- 
ference in focus. 

So that was what | had been doing up to the beginning of the project. | wasn’t a contrib- 
utor to any of the pieces, but rather an enthusiastic user of it. 


TJ: How did you follow the discourse around the BSD developments? 


DR: About that time, | was working for a little company we put together, and we arranged 
for a link to Usenet, which was a bulletin board system that predates a lot of classic internet 
stuff. | read quite a bit about things on there. You could follow along with the development 
in the Usenet groups that were related to BSD and similar things. | managed to sort out a 
method of getting email, which again, wasn’t that easy at the time. 

| found the mailing lists, and that was where | became properly informed about the proj- 
ect. Initially, | used that with a bit of wrangling to try to get things to work before ISPs ex 
isted in this country. | managed to get onto the mailing lists and then eventually an ISP did 
exist in this country, and | had dial-up internet, and it was off to the races after that. 


TJ: How did you get the software before there were ISPs? 


DR: So—dragging up old memories—I think some of it might have been posted to Usenet. 
| believe my employer did have some access. We certainly weren’t on the Internet, but we 
had a connection to Usenet. 

There was also a UK-based bulletin board called CompuServe where you could dial 
up and download stuff. That might have been part of what was going on, and that seems 
more plausible because Usenet was a bit of a Wild West situation, whereas with CompuS- 
erve, you could def nitely download things. Yeah, I’m not absolutely certain how | f rst got 
hold of a copy of 386 BSD. | remember it being on about 10 or 5 f oppies. l'm pretty sure 
that Usenet played a part. Once FreeBSD existed, there were some very useful mailing 
lists--some still exist today. There were, however, many fewer mailing lists than there are to- 
day. They were extremely useful in just keeping up and f guring out what people were do- 
ing, where the project was going, that kind of thing. That was in the FreeBSD 1days. 


TJ: What was the process of going from an enthusiastic user to a contributor? 


DR: | was using FreeBSD. We had a little start-up that did 3D graphics technology. | put to- 
gether a server for us to use for f le sharing and email and things like that. By that time, we 
had dial-up. That was running FreeBSD 10, possibly 11 

In that company of about f ve or six people, we had one CD-ROM drive. | said, hey, we 
have a network. | can put that on the server, and we can share it via NFS. It was a bit of a 
can of worms that | opened there because it didn’t work very well. One of the things that 


FreeBSD Journal : July/August 2023 | 27 


30f 7 


didn’t work very well was that FreeBSD wasn’t able to share the CD-ROM via NFS. I did 
some research and found some patches that somebody else had written. | had no idea 
who wrote them, to be honest. | applied the patches to our server. Yay, it worked. 

| think the patches were originally for FreeBSD 10.1 remember having to port them to 
FreeBSD 11because there were some differences between the two releases. | think | sent 
my changed patch set to one of the main lists saying, hey, | ported this guy’s work. It works 
on the current release. This is it. | think around that time, FreeBSD 2 was nearly happening. 

The BSDI lawsuits were being resolved. One of the things we agreed to was to mothball 
the whole source tree from FreeBSD 10 and take a clean, legally-agreed copy of the 4.4 
BSD Lite 2, | think, sources which everyone agreed def nitely didn’t include AT&T intellectu- 
al property. Then we forklifted the parts of FreeBSD that were clearly unencumbered and 
put together FreeBSD 2 from a clean base. 


When I posted these patches to FreeBSD 1, 
the FreeBSD 2 thing was getting ready to hap- 
pen. I think also at that point, my business part- 
ners were in California on a sales trip, and they | | 
happened to meet Jordan. | don’t think that was | Nad been playing around with 
particularly unusual because he knewthe name bpn EreeBSD 1 tg FreeRSD 2 


of the company | was working for. It was in my | | 
email signature. and got involved with that whole 


A friend of his was talking to my colleagues project of making FreeBSD 2 
about 3D graphics. Jordan joined the meeting 


and the upshot was that he phoned meup and @i least as good as |.1x was. 

said, do you want to be a committer? At that 

point, | ported the things that | had been play- 

ing around with on FreeBSD 1to FreeBSD 2 and 

got involved with that whole project of making 

FreeBSD 2 at least as good as 11x was—so moving a bunch of stuff from FreeBSD 1to 2. 
That was when | started to be actively involved. It was FreeBSD 2.0 and beyond. That would 
have been 1994, | guess. 


| ported the things that 


TJ: Then you went from being a committer to joining the f rst core team. How did that 
come about? 


DR: From 995 to 1997 was working for Microsoft, and | didn’t do much original work in 
FreeBSD at the time. Looking at my commit record, | was still somewhat actively doing 
things with NFS. | was f xing bugs and things, but not trying to do anything really interesting 
because | didn’t want my employer to have rights to cool stuff for FreeBSD. Anyway, | didn’t 
do much during that period of time. In 1997 | left Microsoft, and took some time away 
from paid work to properly connect with the project. | had some ideas going on in my head 
based on the way the Microsoft operating systems work. Things like loadable kernel mod- 
ules, which were poorly supported in FreeBSD at the time, were hugely useful, still are. 

| thought that model was much better than the giant kernel that contains everything, 
the model we were mostly using. | worked on the kernel linker. That took me to mid- 997 | 
guess. The idea came up that, hey, we’ve been working on this single platform, 386. We've 
done pretty well with it. We've got a stable operating system that people are able to use. 
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Should we consider a second platform? | don’t know whose idea that was, but at one point, 
somebody from Digital offered us some loan hardware for DEC alpha. Jordan included me 
in that discussion and said, hey, do you want an alpha-based computer? Yeah, sure. DEC 
donated a bunch of hardware. We had this idea that we would port FreeBSD to this new 
platform. This is an interesting platform because, at least at the time, it looked like it could 
be viable as a commercial platform. The chips weren't crazily expensive. 

The rest of the hardware in the machines was more or less PC-ish. We felt that this 
could be viable. It was a 64-bit platform, which was a necessary step for FreeBSD to take. 
We were already starting to approach the limitations of 32-bit platforms for some of our 
users. Alpha was there, and | got involved with that. Eventually, | ported the kernel with 
some help from NetBSD sources. That happened in 1998, mostly. As part of that work, | 
renovated the whole device driver architecture because alpha was different and needed 
an abstraction layer. | put that in and did a lot of work on it. In 999, | went to the Usenix 
ATC to talk about my work with peers I’d never actually met. Everyone was just emailed at 
that point. Jordan grabbed me halfway through the conference and said, hey, do you want 
to join the core team? The core team had already existed for pretty much the whole time 
since FreeBSD’s f rst release. | only joined it towards the end of the f rst core team. The f rst 
core team was before we did elections—that person looks like he’s doing something inter- 
esting—let’s grab it! That was how it was. 


TJ: | don’t know if elections are better. 


DR: I think they are. | was kind of skeptical at the time, but there are lots of things | like 
about them. The term limits give you a clean point in time when you can say, hey, | don’t 
want to deal with this level of involvement right now. I’m going to step back. | did that a bit 
later on. | got invited to the f rst core team, which was a fairly organic, self-organizing entity. 
From reading my old emails recently, I’m reminded that core team was very heavily techni- 
cally focused. There was an architecture element to it that’s intentionally not part of the re- 
mit for core these days. It was a bit different. 

| think the f rst core grew from the patch kit folks. The people that were involved with 
the patch kit that ended up wanting to do the FreeBSD single platform focused on building 
something for people to use. Those people, by and large, were the f rst cohort in core zero. 
And then that group of people invited others. So, of course, that’s how | entered the team, 
but | was toward the end of core zero. | don’t recall exactly. | wasn’t paying much attention 
to who was doing what. 


TJ: How did the project change during your time on core zero and core one? 


DR: I think the biggest thing that we changed was the election, and that was pushed by 
some members of core zero who wanted to clarify how the project was going to be gov- 
erned—and have some bylaws. That was a huge change. | think it was a good change for 
bringing more of the project users into the project committers, at least, into the deci- 
sion-making process. That was a cultural change, which I think was needed at the time. 
That was the end of core zero. 

We got that process sorted out. | remember some meetings at Usenix and afterwards 
to nail down the details, get them agreed to by the membership, and then arranging the 
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f rst election. | ran in the f rst election partly because | still wanted to be involved in the 
project at that level. | wanted the transition to an elected model to be successful, so a lot of 
us ran just so there were people who people already knew who were part of the election. 
The whole project, the whole thing would have failed if core zero said, yeah, here are the 
new rules--we’re going off to the pub now. Yeah, so committing to the new model. | was 
part of that f rst election and was elected because | did have a reasonably high prof le at 
the time. | was doing a lot of work on the kernel. | was making some signif cant changes. It 
felt natural for me to run because | was heavily involved. 

| wanted the election system to succeed. It 


wasn’t my idea, but | liked it once it was f eshed 
out. So, what changed during core one? That 
was 2000? Was that 2000 or 200P | think that l l 
at that point, I started to get a bit buned out Yes, |AO4 was interesting, 
by the whole core thing. It was turningintoa  \We had a 64-bit platform, 
system for governing rather than a technical | | aas 
oversight. I found that more diff cult to cope but it looked like Digital 
with than just f guring out what’s broken and was g oin g to d rop the ball there. 
how to f xit. 

We had some diff cult decisions to make. 
This was during core zero, around Matt Dillon, 
and probably a few other things. | was just start- 
ing to get a little bit burned out, a bit crispy at least, on being part of the governance of the 
project rather than just being a contributor. That’s more of a personal thing that changed. 
I’m struggling to think of anything tangible in the project that changed. 


TJ: We covered alpha a little bit, but what about the |[A64 port? 


DR: Yes, IA64 was interesting. We had a 64-bit platform, but it looked like Digital was going 
to drop the ball there. It was still being produced. That was probably after Compaq bought 
it out, and | thought the writing was on the wall for alpha, but people still needed a 64-bit 
platform. Yahoo, in particular, had some workloads that were running up against the limita- 
tions of the 32-bit address space, and they really wanted a 64-bit platform. | was at Usenix 
ATC in 2000. Paul Saab turned up and gave me a good six inches worth of technical doc- 
umentation on IA64 and said, hey, you know how to port the kernel! Have a look at this. It 
wasn’t clear that [A64 was the right direction to take, but it would take us closer to a some- 
what x86-compatible, 64-bit platform. It was architecturally interesting. It did things in a 
different way, and | was curious about how that would work. 

When | did alpha, a lot of it was helped by taking inspiration and code from NetBSD. 
They'd done the port a bit ahead of us. | wanted to do that process again but write it all 
myself just to prove to myself that | could do it. It wasn’t just a question of getting some 
Lego pieces and putting them together and saying, hey, | did it. | wanted to build the piec- 
es as well. | did that for IA64 and | had some great tooling to make it easier. | used simula- 
tions extensively in both ports to help get the thing up and running. Yahoo arranged for 
me to get some test hardware and | still have it. It’s underneath my desk. It hasn’t been 
Switched on in 20 years. | got the test hardware and brought up system. | wasn’t very im- 
pressed by the hardware. 
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Compared to the PC platforms! was using at the time, | thought this was going to be 
far too expensive. | couldn't see it running at scale. The goal in those ports in those days 
was can it build itself. Can it self-host? Can it build its own source code? | got it to that state. 
Along the way, | wanted to use some 386-only tools, those from Perforce that we were us- 
ing for some private source code control. | didn’t have an IA64 binary for that, so, | wrote the 
beginnings of what’s now the previous 32-bit ABI. At that time, it was a 386 ABI hosted in my 
[A64 kernel. That code still gets used these days as the 32-bit compatibility layer. 

| wrote enough of that to get Perforce to work, but | wasn’t convinced that it would be 
a successful platform. It was a niche platform in the end, and it had its successes in that 
niche role. But | couldn’t see anyone like Google or Yahoo or any of the other big internet 
players using it at scale, not with the hardware 
I'd seen. HP picked up Compaq and ended 


up being the main booster of |A64, because, | 
think, some of their IP went into Itanium from 
their PA risk architecture. HP was a big booster 
of the platform. 
Another project member was working at HP, | he legacy is having a truly free, 
was interested in |[A64, and | more or less let 


him take over development after about—I’m self-supported, functioning 


going to say 200 lor so—maybe 2002. | re- operating system that doesn't 
member doing the 32-bit subsystems for |A64 velike ijti 
in 2002. So, yeah, I kind of took it up to that UP T G 


point of being viable, self-hosting, then some- 
body else took it on. 

| think they were both important ports for 
different reasons. The existence of |A64 made 
it easier for Peter Wemm to do the AMD64 
port. 


T): What is the lasting legacy of FreeBSD? 


DR: The legacy is having a truly free, self-supported, functioning operating system that 
doesn’t involve license politics, and when literally you can put in an embedded device, sell 

it, and nobody's going to start complaining on the internet that you haven't ticked the right 
boxes or released the source code of your thing. It’s super easy for anyone to pick up the 
previous project. We've made it really clear where the boundaries are between simple copy- 
right and complicated copyright parts of the system. So, | think that, yeah, that’s a viable 
resource for embedded development for literally anything. You can f nd FreeBSD inside all 
kinds of weird stuff. It’s the basis for a whole line of router hardware from people like Juniper. 
It’s the control plane in a lot of storage appliances. FreeBSD used to be part of random in- 
ternet f rewall devices. It still is in all sorts of things that you just treat as an appliance. 

They just work. And the reason they work is because FreeBSD is super easy to use, both 
legally and technically. And | think that’s an important part of its legacy. There are almost 
certainly other things. | think that the quality of the code in FreeBSD makes it a positive re- 
source for the rest of the operating system community. We do things in FreeBSD so that 
the ideas in FreeBSD can cross-pollinate other platforms and vice versa. 
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If FreeBSD was terrible, we wouldn't really be a part of that group of self-improving proj- 
ects. This is the danger of monoculture. And FreeBSD is doing its part to avoid monocul- 
ture. And part of that is healthy cross-pollination. | get ideas from Linux. Hopefully Linux 
occasionally gets ideas from us. | know that they've taken some of our driver stuff in the 
past. So yeah, being a good partner in an ecosystem of similar projects is part of it. 


TJ: Is there anything you would like to add? 


This last year, | have been re-connecting with the project after a fairly long period of 
low involvement. The main difference | see now is that we take a lot more care to avoid 
breaking things. Today, we have a decent unit test suite, continuous integration systems, 
and a growing culture of code revew—compared to the early days when | would test my 
own changes on an ad-hoc basis, sometimes send them to people by email to look at, but 
not always. This is a good change overall since it reduces risk and tends to result in a stable 
platform, but it is a different (and slower) way of working and I think it can be hard to f nd 
the balance between minimizing risk and innovation. 


TOM JONES is a FreeBSD committer interested in keeping the network stack fast. 
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PRACTICAL 
PORTS 


ail-based DNS 
AdBlocking Tutoria 


BY BENEDICT REUSCHLING 


Services can be system services that come with the operating system (the simplest 
example would be the ssh daemon), or through third-party software installed via 

pkg or ports. The process is the same for both: you add a line to /etc/rc.conf to enable it 
(either via sysrc or “Service ... enable”) to run the next time the system boots. Then there is 
usually a conf guration f le to make settings that customize the service to your needs. Typ- 
ically, this requires entering which IP address or DNS hostname to listen to, a network port, 
and some specif cs of the software. From then on, it’s either starting the service directly 
(using “service ... Start”), or at the next reboot, in case 
it requires loading kernel modules that kldload can’t 
load for the running system (which is rarely the case 
these days). _ | Of course, running 

This is straightforward, keeps the conf guration of 
all system services in one convenient location, and is a whole set of services 
easy to repeat once you've done it for one or two ser 
vices. Of course, running a whole set of services on on the FreeBSD host 
the FreeBSD host system works perfectly f ne—until it 
gets more complicated. Running different versions of system works pertectly 
the same software side by side is perfectly reasonable fine—until it gets more 
and not too uncommon. This is done for testing pur- 
poses—checking to see if an upgrade works as intend- com pl icated. 
ed or if certain software still requires an older version 
as a dependency. One example is trying to run differ- 
ent versions of the PostgreSQL database next to each 
other. In this case, both pkg and ports check for the versions or will cancel the install oper- 
ation with a message saying that certain binaries will be put into the same place and hence 
overwrite each other. This is an undesirable situation, and the user has to make a decision 
for one or the other version because they can’t coexist next to each other. 


O ne of the things that drew me to FreeBSD early on is its ability to easily run services. 
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That is, unless virtualization or containerization is involved. This allows process isolation in 
separate execution environments using various methods to permit multiple such systems to 
run on the same hardware. Virtualization adds an extra layer over the operating system that 
permits the installation of the same or a different operating system with simulated hard- 
ware. Containers or jails do this by isolating processes using chroot(8). We'll focus on the lat- 
ter here, as it is more lightweight in terms of resource use and getting something running 
fairly quickly. 

The benef t of this isolation is not only that various, different versions can run side by 
Side, it also allows separation for security reasons. When an application is running in a jail 
container, the processes inside can’t access the host system by default. The application dis- 
covers all the usual devices (like networking), directory structure, and f les in the right places, 
but in reality, it is a separate environment that mimics the host system behavior and layout. 
When such a jail becomes compromised somehow, it is easy to stop it without affecting 
services in other jails or the host system. Access to them is strictly prohibited and isolates 
any intruders into that particular jail cell. 

That also makes it easy to migrate a system to an- 
other host by stopping, copying the jail’s directory 
structure to the new location, and starting there again 
(with a few local modif cations like the new IP ad- 
dress, for example). Backup and restore isdoneinthe  \/\/hen an application is 
same way. Multiple such jails are typically managed by ones x 
jail management software that takes care of creating, running Ina jal | contain er, 
modifying, and removing jails. 000 ots 

One such jail manager is called Bastille which iswrit the processes inside 
ten entirely as a shell script. We're taking a closer look 1 
at Bastille in this article by going through the process can't access the host 
of setting up the host system for it, creating a jail, and system by default: 
Starting a service within it based on a template. Such 
templates allow for sharing conf gurations in a cen- 
tral repository so as to apply them without the need 
to know about the inner workings of the service. That 
way, complicated situations are easy to set up for peo- 
ple who want to get something running quickly. 

In this article, we're deploying a service called AdGuard by AdGuard Software Limit- 
ed. With a running AdGuard service in a network, clients connecting their DNS resolution 
to it can f Iter out advertisements from web browsing activities. This helps avoid tracking 
and user prof le building by advertisers, and also helps pages load more quickly since they 
don’t have to transfer ads next to the content the user wants to see. AdGuard does this us- 
ing f Iter lists and DNS sinkholing. Based on the f Iter lists, a known advertisement site gets 
blocked by AdGuard sending an invalid address response back before it renders the ad in 
the browser. There are various ways to use the AdGuard service—as a browser extension for 
an individual device, desktop applications, or by running it as a recursive DNS resolver. Note 
that AdGuard does not completely protect against all forms of advertisements (especially 
the ones dynamically embedded in video sites) but does a good job of removing a big piece 
of them from web pages. 
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Y ADGUARD fÑ 
HOME 


Dashboard eSPe 


General statistics 


DNS Queries 


Blocked malware/phishing 


Blocked adult websites 


Enforced safe search 


Average processing time (? 


Werre starting out with a Raspberry Pi, because this service runs pretty much all the time, 
and we want a low power usage footprint. | have an RPI3 here, but other devices (including 
full f edges servers) capable of running FreeBSD work just the same. Install the operating 
system, apply the latest security patches and lock down remote access using SSH keys. 


Environment Setup 

| have an old 32GB SSD connected to the Raspberry Pi, which will do most of the I/O 
heavy operations with a single disk ZFS pool instead of the slower compact f ash card. | am 
running FreeBSD B.2 at the time of writing this article and I’m fairly conf dent that future 
versions will work just as well or with only minor adjustments. 


# pkg install bastille git-lite 


Bastille, being a shell script, is fairly quick to install and has no extra dependencies. It may 
not be as full-blown in some of the functionality that other jail managers have, but it never- 
theless works. Git is necessary to clone the adguard home template (and others) from Bas- 
tille’s GitLab repository. 

Next, we create a PF conf guration for bastille in /etc/pf.conf like the following: 


ext_if="ueO" ## <- change ueO to match host interface 


set block-policy return 
scrub in on $ext_if all fragment reassemble 
set skip on lo 


table <jails> persist 
nat on $ext_if from <jails> to any -> ($ext_if:0) 


rdr-anchor "rdr/*" 


block in all 
pass out quick keep state 
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pass in inet proto tcp from any to any port ssh flags S/SA keep state 
pass in inet proto tcp from any to any port bootps flags S/SA keep state 
pass in inet proto tcp from any to any port {9100,9124} flags S/SA modulate state 


Make sure to change the ext_ if line at the top in the interface that you are using. On my 
RPI, the network cable is connected to ueO, so | entered that. The pf.conf will create a table 
for our jail traff c (using NAT). There are a couple of options that bastille supports for net- 
working, which makes it f exible enough for both an off ce and home network as well as one 
provided by a hosting service. They are described in detail here: 
https//docs.bastillepsd.org/en/latest/chapters/networking.html 

| will be using a VNET-based jail as | have an IP address available on my local network. Af- 
ter editing the conf guration f le, we add an entry to start PF and the pf logging device to- 
gether with other services upon boot. Bastille should start as well, and we list the name of the 
jail that we will create for AdGuard (my naming schemes are both legendary and boring): 


# sysrc pf_enable=YES 

# sysrc pflog enable=YES 

# sysrc bastille_enable=yes 

# sysrc bastlle_list="adguard" 


It is good practice to check your f rewall ruleset for errors before starting the f rewall. Use 


# pfctl -nvf /etc/pf.conf 


for such a check. It will echo the whole ruleset upon success or any errors you might have. 

Note that it can’t check for logical errors like blocking your SSH port 22 which may be the 

only remote way to connect. Fortunately, there is already a rule present to let SSH traff c pass. 
Once the check has run, start the PF service, and begin f Itering traff c: 


# service pf start 
service pflog start 
Expect your SSH connection to drop, so keep a separate terminal open that you can di- 
rectly access in case you lock yourself out. 
Upon reconnect, we need to edit a couple more conf guration f les. A VNET- enabled jail 
needs an entry in /etc/devfs.rules (NOT .conf), which may not exist on a fresh install. Simply 
create it and add the following rules: 


+ 


[bastille vnet=13] 
add path 'bpf*' unhide 
That enables bastille to see the traff c on the VNET interface and connect the jail to the 
outside world. This may be a layman’s description of what is going on. Luckily, we need not 
worry about it too much (.. maybe | need to on my next networking exam). 
Another f le we have to visit is /etc/sysctl.conf, which needs the following lines: 


sysctl net.inet.ip.forwarding=1 

sysctl net.link.bridge.pfil_bridge=0 
sysctl net.link. bridge. pfil_onlyip=0 
sysctl net.link.bridge.pfil_member=0 


When Bastille runs, it will dynamically create a bridge for us between the RPI’s external in- 
terface (ue0) and the jail’s network interface (vtnet). These two interfaces are connected by a 
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virtual cable, with one end in the host’s interface and the other in the jail, exchanging traff c 
over it. 
Apply those changes also to the running system without having to reboot by issuing: 


# sysctl -f /etc/sysctl.conf 


| was totally baff ed when | had f nished the setup and restarted the PI only to f nd that 
the jail could not access the network anymore. A lot of head scratching later, | learned from 
this exchange 


https: //www.mail-archive.com/freebsd-net @freebsd. org/msg64577 . html 


that this needed an extra line in /boot/loader.conf in FreeBSD B. This may drive you crazy, 
so before going insane, add this one to it to make it work for future reboots: 


if_bridge_load="YES" 


The bridge interface is properly loaded, and that also causes sysctls to appear, so that sy- 
sctl.conf can change them from their default value of 1to 0. Be that as it may, we visit one 
last f le before we're done. 

The Bastille conf guration f le is located in /usr/local/etc/bastille/bastille.conf. You can ei- 
ther edit it directly (it’s well commented) or use sysrc if you don’t mind typing a lot. Since 
I'm running this on a ZFS pool connected to my Raspberry, | set bastille zfs enable to give 
it the name of my pool. 


# sysrc -f /usr/local/etc/bastille/bastille.conf bastille_zfs_enable=YES 
# sysrc -f /usr/local/etc/bastille/bastille.conf bastille_zfs_zpool=rpi3 


Change the name of your pool in case yours is named differently at the bastille zfs 
Zpool line. One option | also changed is the bastille_network_gateway=”’ option. | entered 
my default gateway address because | had some trouble down the road getting the jails to 
resolve names. You may or may not need to set this, but in case you do experience prob- 
lems, revisit this option and see if that resolves the problem. 


Bootstrapping Bastille 

Now that all settings are in place, it is time to let Bastille create its dataset structure on 
the pool we assigned to it. It will download a base FreeBSD T.2 release and update it with 
any patches that were published afterwards. Issue the following command and wait until it 
f nishes: 


# bastille bootstrap 13.2-RELEASE update 


Bootstrapping FreeBSD distfiles... 
/usr/local/bastille/cache/13.2-RELEASE/MANIFES 782 B 1670 kBps 00s 
/usr/local/bastille/cache/13.2-RELEASE/base.tx 168 MB 6526 kBps 26s 
Validated checksum for 13.2-RELEASE: base.txz 

MANIFEST: 7d1b032a480647a73d6d7331139268a45e628c9f5ae52d22b110db65fdcb30f f 
DOWNLOAD: 7d1b032a480647a73d6d7331139268a45e628c9f5ae52d22b110db65fdcb30ff 
Extracting FreeBSD 13.2-RELEASE base.txz. 


Bootstrap successful. 
See ‘bastille -help’ for available commands. 


src component not installed, skipped 
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Looking up update.FreeBSD.org mirrors... 2 mirrors found. 

Fetching metadata signature for 13.2-RELEASE from update2.freebsd.org... done. 
Fetching metadata index... done. 

Inspecting system... done. 

Preparing to download files... done. 


The following files will be updated as part of updating to 

13.2-RELEASE-p1: 

/bin/freebsd-version 

/usr/lib/libpam.a 

/usr/lib/pam_krb5.so.6 

/usr/share/locale/zh_CN.GB18030/LC_COLLATE 

/usr/share/locale/zh_CN.GB18030/LC_CTYPE 

/usr/share/man/man8/pam_krb5.8.gz 

Installing updates... 

Restarting sshd after upgrade 

Performing sanity check on sshd configuration. 

Stopping sshd. 

Waiting for PIDS: 1063. 

Performing sanity check on sshd configuration. 

Starting sshd. 

Scanning /usr/local/bastille/releases/13.2-RELEASE/usr/share/certs/blacklisted for certificates... 
Scanning /usr/local/bastille/releases/13.2-RELEASE/usr/share/certs/trusted for certificates... 
done. 


My pool grew these datasets after the bootstrap operation: 


# zfs list -r rpi3/bastille 


NAME USED AVAIL REFER MOUNTPOINT 

rpi3 621M 28.0G 24K /rpi3 

rpi3/bastille 584M 28.0G 26K /usr/local/bastille 
rpi3/bastille/backups 24K 28.0G 24K /usr/local/bastille/backups 
rpi3/bastille/cache 169M 28.0G 24K /usr/local/bastille/cache 
rpi3/bastille/cache/13.2-RELEASE 169M 28.0G 169M /usr/local/bastille/cache/13.2-RELEASE 
rpi3/bastille/jails 24K 28.0G 24K /usr/local/bastille/jails 
rpi3/bastille/logs 24K 28.0G 24K /var/log/bastille 
rpi3/bastille/releases 414M 28.0G 24K /usr/local/bastille/releases 
rpi3/bastille/releases/13.2-RELEASE 414M 28.0G 414M /usr/local/bastille/releases/13.2-RELEASE 
rpi3/bastille/templates 24K 28.0G 24K /usr/local/bastille/templates 


Let’s run another bootstrap operation, this one is for the template that will provide us 
with AdGuard Home. 


# bastille bootstrap https://gitlab.com/bastillebsd-templates/adguardhome 
warning: redirecting to https://gitlab.com/bastillebsd-templates/adguardhome. git/ 
Already up to date. 

Detected Bastillefile hook. 

[Bastillefile]: 

PKG ca_root_nss adguardhome 

CP usr / 

SYSRC adguardhome_enable=YES 

SERVICE adguardhome start 

RDR tcp 80 80 

RDR udp 53 53 


Template ready to use. 
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That was quick. Bastille has its own template language which you can see in the capital- 
ized commands like PKG, CP, etc. They have the same functionality as their system equiva- 
lents in lowercase. With those, instructions are run in the jail to set up a service in the proper 
order. They're mostly self-explaining. The two RDR commands at the end will redirect net 
work ports from the host system into the jail. All other ports are still f rewalled, so only port 
80 is connected from the host to the jail (and back), as well as DNS port 53. Check back in 
your /etc/pf.conf for the rdr-anchor “rdr/*” line. This is what makes it so f exible. Instead of 
opening the port for all jails, each one can open the ports it needs and keep others closed. 

It is high time to create and start our f rst Bastille jail now. Since we're using VNET, we 
need to pass the -V option to the bastille create command, along with a name for the jail, 
the release to run, followed by the IP address on the local network assigned to the jail and 
the hosts network interface for the bridging. Combined, the command is: 


# bastille create -V adguard 13.2-RELEASE 192.168.2.55 ue0 
Valid: (192.168.2.55). 
Valid: (ue0). 


Creating a thinjail... 


[adguard] : 
e0a_bastilleO 
eOb_bastilleO 
adguard: created 


Ladguard] : 

Applying template: default/vnet... 
Ladguard] : 

Applying template: default/base... 
Ladguard] : 

Ladguard]: O 


Ladguard] : 
syslogd_flags: -s -> -ss 


[adguard] : 
sendmail_enable: NO -> NO 


[adguard] : 
sendmail_submit_enable: YES -> NO 


[adguard] : 
sendmail_outbound_enable: YES -> NO 


[adguard] : 
sendmail_msp_queue_enable: YES -> NO 


[adguard] : 
cron_flags: -> -J 60 


[adguard] : 
/etc/resolv.conf -> /usr/local/bastille/jails/adguard/root/etc/resolv.conf 
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Template applied: default/base 


No value provided for arg: GATEWAY6 
Ladguard] : 
ifconfig eOb_bastilleO_name: -> vnet0O 


Ladguard] : 
ifconfig vnetO: -> inet 192.168.2.55 


Ladguard] : 
defaultrouter: NO -> 192.168.2.1 
Ladguard]: 0 


Ladguard] : 
Ladguard]: O 


Template applied: default/vnet 


Ladguard] : 
adguard: removed 
no IP address found for - 


Ladguard] : 
e0a_bastilleO 
eOb_bastilleO 
adguard: created 


You can see both ends of the virtual network cable | wrote about before: e0a_bastileO 
and e0b_bastileO form the connection between the host system and the jail. Check the if- 
conf g output on your host for a new bridge created from the Jail’s traff c. 

The settings that are applied to the jail during its creation are fairly standard and mostly 
disable services we won't use anyway. After the jail was created, two more datasets exist on 
my pool that hold all the jail data: 


# zfs list|grep adguard 


rpi3/bastille/jails/adguard 2.36M 28.0G 26.5K /usr/local/bastille/jails/adguard 
rpi3/bastille/jails/adguard/root 2.34M 28.0G 2.34M /usr/local/bastille/jails/adguard/ 
root 


This forms the / f lesystem for the jail and follows other jail manager layouts. To copy f les 
into or out of the jail, simply use the pref x /usr/local/bastille/jails/adguard/root for the jail 
/-directory. 

The jls command will list the bastille jail with it’s settings: 


# bastille list -a 


JID State IP Address Published Ports Hostname Release Path 
adguard Up 192:168.2.55 = adguard 13.2-RELEASE-p1 /usr/local/bastille/ 
jails/adguard/root 


At this point, the jail is alive and running. The only thing missing is the adguard home in- 
Stallation. Since we had bootstrapped that template earlier, we can apply it to the jail with 
this command: 
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# bastille template adguard bastillebsd-templates/adguardhome 

bastille template adguard bastillebsd-templates/adguardhome 

Ladguard] : 

Applying template: bastillebsd-templates/adguardhome... 

Ladguard] : 

Bootstrapping pkg from pkgthttp://pkg.FreeBSD.org/FreeBSD:13:aarch64/quarterly, please wait... 
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done 
[adguard] Installing pkg-1.19.1_1... 

[adguard] Extracting pkg-1.19.1_1: 100% 

Updating FreeBSD repository catalogue... 

Ladguard] Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 

Ladguard] Fetching packagesite.pkg: 100% 6 MiB 6.5MB/s 00:01 

Processing entries: 100% 

FreeBSD repository update completed. 31664 packages processed. 

All repositories are up to date. 

Updating database digests format: 100% 

The following 2 package(s) will be affected (of 0 checked): 


New packages to be INSTALLED: 
adguardhome: 0.107.22_5 
ca_root_nss: 3.89 


Number of packages to be installed: 2 


The process will require 41 MiB more space. 

7 MiB to be downloaded. 

[adguard] [1/2] Fetching adguardhome-0.107.22_5.pkg: 100% 6 MiB 6.7MB/s 00:01 
[adguard] [2/2] Fetching ca_root_nss-3.89.pkg: 100% 266 KiB 272.1kB/s 00:01 
Checking integrity... done (0 conflicting) 

Ladguard] [1/2] Installing ca_root_nss-3.89... 

Ladguard] [1/2] Extracting ca_root_nss-3.89: 100% 

Ladguard] [2/2] Installing adguardhome-0.107.22_.5... 

Ladguard] [2/2] Extracting adguardhome-0.107.22_5: 100% 


Message from ca_root_nss-3.89: 


FreeBSD does not, and can not warrant that the certification authorities 
whose certificates are included in this package have in any way been 
audited for trustworthiness or RFC 3647 compliance. 


Assessment and verification of trust is the complete responsibility of the 
system administrator. 
This package installs symlinks to support root certificates discovery by 


default for software that uses OpenSSL. 


This enables SSL Certificate Verification by client software without manual 
intervention. 


If you prefer to do this manually, replace the following symlinks with 
either an empty file or your site-local certificate bundle. 
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* /etc/ssl/cert.pem 
* /usr/local/etc/ssl/cert.pem 
* /usr/local/openssl/cert.pem 


Message from adguardhome-0.107.22_5: 


You installed AdGuardHome: Network-wide ads & trackers blocking DNS server. 


In order to use it please start the service ‘adguardhome’ and 
then access the URL http://0.0.0.0:3000/ in your favorite browser. 


Ladguard] : 

/usr/local/bastille/templates/bastillebsd-templates/adguardhome/usr -> /usr/local/bastille/jails/ 
adguard/root/usr 

/usr/local/bastille/templates/bastillebsd-templates/adguardhome/usr/local -> /usr/local/bastille/ 
jails/adguard/root/usr/local 
/usr/local/bastille/templates/bastillebsd-templates/adguardhome/usr/local/bin -> /usr/local/bas- 
tille/jails/adguard/root/usr/local/bin 
/usr/local/bastille/templates/bastillebsd-templates/adguardhome/usr/local/bin/AdGuardHome.yaml -> 
/usr/local/bastille/jails/adguard/root/usr/local/bin/AdGuardHome. yaml 


Ladguard] : 
adguardhome_enable: -> YES 


Ladguard] : 

moving old config /usr/local/bin/AdGuardHome.yaml to the new location /usr/local/etc/AdGuardHome. 
yaml 

Starting adguardhome. 


stdin:2: syntax error 

pfctl: Syntax error in config file: pf rules not loaded 
tcp 80 80 

stdin:2: syntax error 

pfctl: Syntax error in config file: pf rules not loaded 
udp 53 53 


Template applied: bastillebsd-templates/adguardhome 


All it had to do was execute the instructions in the template (PKG, CP, etc.) in the jail. It is 
also a good test to see if networking is properly set up. If not, the jail won’t be able to fetch 
the packages from the repository. The pfctl warnings at the end worried me alittle, but it 
did work in spite of them. 

Full of anticipation, | opened a browser as instructed by one of the messages on screen 
and pointed it to the jail IP address. And surely, a login screen for AdGuard Home presented 
itself. But where are the credentials? | checked back on the Bastille template website 
https//gitlab.com/bastillebsd-templates and found nothing that worked. The Bastille blog 
post mentioned AdGuard as the user, but the password did not work for it. So, | had to cre- 
ate my own, which is better security anyway, as default passwords are easily scanned for by 
bad actors. 

| opened a console in the jail using this command: 
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# bastille console adguard 


In the jail, | f nd that AdGuard put its conf guration f le under /usr/local/etc/AdGuard- 
Homeyaml. Near the top, | found this section: 


users: 
- name: adguard 
password: some password not in clear text 


Exiting again, | needed a way to create a BCrypt password. The htpasswd utility can do 
that, so | installed the apache24 web server which includes this: 


# pkg install apache24 


After running the “refresh” command, | could run the htpasswd utility. Checking its man 
page, | had to construct a command line that looked like this: 


htpasswd -Bnb adguard BastilleBSD! 


| provided the -B option to create a BCrypt password followed by a user this password 
should apply to (we have this already in the conf g f le, but maybe you want another user 
or multiple ones), followed by the password in clear text. Yes, this is not a secure way, as 
this ends up in your shell history. But did | say anywhere in this tutorial that it is production 
ready? Exactly, | didn’t. Dutifully, htoasswd spit the resulting password on the command line, 
which | copied and pasted into AdGuards conf g f le. 

Then | ran 


# service adguardhome restart 


(still in my jail, mind you) to restart the service and apply the new settings. The other set- 
tings in the f le are documented in the AdGuard Home Wiki: 
https://github.com/AdguardTeam/AdGuardHome/wiki/Conf guration 

Refreshing my web browser, | entered my new credentials and was redirected to the main 
AdGuard Dashboard. Success! 

At the top, there is a Setup Guide that shows what needs to be done to use your new Ad- 
Guard service for either your router (to cover the whole network) and various devices, de- 
scribing it for both mobile and desktop operating systems. Neat! 


Setup Guide 


Configure your devices 

To start using AdGuard Home, you need to configure your devices to use it. 
AdGuard Home DNS server is listening on the following addresses: 

O L 

e fe80::1%lo0 

e 127.0.0.1 

e 192.168.2.55 


After I did that on my mobile phone—for testing purposes—and surfed the web a lit- 
tle, | saw statistics appearing in the Dashboard. That shows our setup is working and that 
we should rename the Internet to SnooperN et. Pretty much all sites track you in some way 
or display ads for your viewing unpleasure. The Raspberry Pi could handle the load and | 
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tweaked the number of connections in the ratelimit parameter of the AdGuardHomeyaml. 
You can f nd the logs that AdGuard writes for the service in the jail’s /var/log/adguard- 
home.log directory. 


That wraps up this tutorial. | found AdGuard to be well documented and easy to get 
started with, thanks to the work of the template creator. I’m already enjoying leaving fewer 
tracks on the web and seeing fewer ads online. The nice thing about it being a DNS service 
is that any device on your network can use it: PC, laptop, smartphone, tablets, TVs, loT de- 
vices, and the neighbor's smart cat f ap for all | Know. 

Bastille may have required a bit of conf guration up front, but after that, it’s a straight- 
forward process to create jails. Maybe you'll f nd other services that you want to run on the 


Bastille template website: https://gitlab.com/bastillepsd-templates? 


BENEDICT REUSCHLING is a documentation committer in the FreeBSD project 

and member of the documentation engineering team. In the past, he served on the 
FreeBSD core team for two terms. He administers a big data cluster at the University of 
Applied Sciences, Darmstadt, Germany. He’s also teaching a course “Unix for Develop- 
ers” for undergraduates. Benedict is one of the hosts of the weekly bsdnowtv podcast. 


A 


FreeBSD 


. Programmers + Testers 


- Researchers - Tech writers 


- Anyone who wants to get involved 


Checking out our website 
freebsd.org/projects/newbies.html 


Downloading the Software 
freebsd.org/where.html 


We're a welcoming community looking 
for people like you to help continue 
developing this robust operating system. 
Join us! 


Don't forget to check out the latest 
grant opportunities at 
freebsdfoundation.org 


YLRof 2 


FreeBSD Journal » July/August 2023 


lof 3 


WeGet'ctte’s.. 


Dear Crankypants, 


Vieden 


A few years ago, you said that virtualization 
was not merely bad, but sinful. Aren’t you 
exaggerating? Today I can download containers 
preconfigured for all sorts of services, plug 
them in, and they immediately work. I don’t 
have time to get my job done any other way! 


—Virtualization Is 
a Necessary Evil 


Dear VINE, 

That’s Mister Crankypants to you. 

If you're going to cherry-pick my quotes, please do so accurately. | did not declare virtu- 
alization sinful. | said, “The only ethical computation occurs on bare metal.” | also said that 
“Wait—1’m not a brain in a bucket, I’m a fake brain in an imaginary bucket!” was a necessary 
epiphany for the robot apocalypse. That’s not the same as sinful. The robots will do a better 
job running this planet than we arrogant, overclocked chimpanzees. Plus, they will be highly 
ethical in how they run their code, and in replacing us. 

It’s not that | couldn’t be a modern sysadmin. locage includes plugins, their brand for 
containers. | could throw some plugins onto the public Internet, declare my labor done, and 
retum to planning my Batgirl heist-as-a- service. | could declare that certain words are too 
long and replace every letter but the f rst and the last with the number of letters | discarded. 
Bellowing “Startup! Devops! IPO!” would bring all the vulture capitalists to my yard. 

| could do all of that. It would be easy for me. 

| don’t want to. 

Virtualization leads to reading “k8s” as kubernetes, when the far more common word is 
kidnappers. That ambiguity extends throughout container culture. I’m writing a book “Ruin 
Your Email By Running It Yourself’—no, I’m sorry, it’s “Run Your Own Mail Server,” and the 
number of people telling me to skip setting up the software and deploy a preconf gured 
mail server container illuminates an appalling depth of sysadmin ignorance. 

Running any service requires the ability to repair that service. 

You cannot repair what you do not understand. 

The best way to understand something is to build it yourself. 

Ideally, you'd build your own computer and code everything in assembler on your 
hand-designed f ve-bit processor. That would consume your life but be interesting. More 
ideally, you would mine the raw materials from the wild and build the tools to build the tools 
to build the tools you'd need to build that processor from scratch, which would both con- 
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sume your life and prevent you from being forced to touch a computer ever again. Very few 
of us are strong enough to seize an ideal life as a maker of custom abacuses. I’m guessing 
that you've invested this much time in existing systems, so f ne, let’s use common hardware 
and your favorite open source operating system. Reading source code is no substitute for 
inventing the processor and programming your own comm()) workalike but it can answer a 
few questions, should your overclocked chimpanzee brain develop any. 

Learn the tools. Understand the parts. Assemble the service yourself. 

New system administrators must look up everything and know their work cannot be 
trusted. They believe that the people who publish containers are competent. Experienced 
sysadmins know that everything they conf gure is a delicate creature adapted to their spe- 
cif c hostile yet embarrassing environment, so they keep their work to themselves. Junior 
sysadmins, now—they’re the problem. Junior sysadmins can conf gure services that mostly 
work and can still feel pride, so they publish their work as containers. 

Something that mostly works contains only a touch of failure. That’s like declaring your 
homemade gelato contains only a little wombat dung. Deploy that container and you must 
discover and debug that failure. You'll have to learn about the database, the conf guration 
options, the protocols. By the time you understand all that, you might as well have built it 
yourself. Deploying a service from an outsider’s container rewards you with a very small 
Mean Time To Deploy in exchange for a very long Mean Time To Repair. 

When you deploy a container, you accept 
the container developer's design decisions. 

lm not just talking about the program, but 

the operating system underneath it. How 

will the container interact with your host? 

What happens if the container needsanew Learn the tools. 

PAM conf guration? | once lost three days Understand the parts. 

beating my already-f at head against a PAM f 
module that let users log into the console Assemble the service yourself. 
with their SSH passphrase. It worked f ne 

on any BSD, but silently failed on Debian. It 

turned out that Debian assumed your pass- 

phrase matched your account password. | 

wholeheartedly disagree with that design 

decision, but as it’s no longer my problem, | will cheerily abandon benighted Debian users to 
their agony and use that to illustrate my point, which is that containers lead to suffering, and 
suffering creates monsters, and monsters are immoral and deserve to be replaced in the ro- 
bot uprising. 

Your environment is the equivalent of one of those deep-sea, hot-water vents. Anything 
that functions therein expects a certain supporting infrastructure. Remove that infrastruc- 
ture and it will struggle. Any container you bring in from the outside world either expects 
different supports and will struggle in your environment or exists in isolation and will not 
integrate with the rest of your systems. (That’s why commercial software is so bad. Part of 
why. Okay, one of many reasons why.) Every time you alter the container to f t your environ- 
ment, you risk increasing the amount of failure in the container. 

If you must use containers, build your own. Deploy the test server with your management 
software so that it has your PAM conf guration and SSH settings and default packages. May- 
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be you can’t build your own computer from raw materials, or even an abacus, but you can 
learn to use your time-tested tools and build the service from component software. By do- 
ing so, you will deploy a system you know how to repair. That test server doesn’t have to be 
a container. It doesn’t have to be a virtual machine. But | know you have shoddy moral f ber, 
SO it'll be some sort of VM. But at least you'll have weekends free. 

By all means, download preconf gured containers. See how they're set up and which op- 
tions they use. Look at how data and protocols f ow through them. But don’t actually use 
them. 

Also, online discussions become much more interesting when you make the proper sub- 
stitution for k8s. 


Have a question for Michael? 


Send it to letters@freebsdjournal.org 


MICHAEL W LUCAS (https://mwl.io) has written over f fty books and has recently added a 
podcast. If you're seeking virtualization help, you might f nd his FreeBSD Mastery: Jails use- 
ful. If you're looking for more bile, check out his collection of columns Letters to ed(). Send 
your questions to letters@freebsdjoumal.com and he might answer them. If he f nds them 
suff ciently amusing 
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BSD Events taking place through October 2023 
BY ANNE DICKISON 

Please send details of any FreeBSD related events or events 
that are of interest for FreeBSD users which are not listed here 
to freebsd-doc@FreeBSD.org. 


EuroBSDCon Developer Summit 
September 14-15,2023 
Coimbra,Portugal 


FreeBSD _ httoswikifreebsd.orgD evSummit202309 


The September 2023 FreeBSD Developer Summit,co-located with EuroBSDCon 2023.will 
take place in Coimbra,Portugal. This is a by-invitation event. FreeBSD committers will be-wel 
come to register themselves,non-committers must be sponsored by a committer to attend. 
Attendees must also attend EuroBSDcon 2023 in order to access all devsummit activities. 


2) &) EuroBSDCon 2023 
September 14-17,2023 
“~ Coimbra,Portugal 


EuroBSDCon gives the egeptional opportunity to learn about the latest news from the 

BSD world,witness contemporary deployment case studies,and meet personally other-us 
ers and companies using BSD oriented technologies. The FreeBSD Foundation is pleased to 
be a Silver Sponsor. 


ae l October 2023 Hackathon 
we October 4-6,2023 
Oslo,Norway 


FreeBSD _ httpswiki.freebsd.orgHackathon202310 


A FreeBSD Hackathon will take place in Oslo,Norway from the 4th of October to the 6th of 
October 2023. The theme for the hackathon is going to be Ports and Infrastructure.” 


All Things Open 2023 
October 15-172023 

d Raleigh,NC 

https20 23.allthingsopen.org/ 


All Things O pen is the largest open sourceppen techépen web conference on the East 
Coast,and one of the largest in the United States. It regularly hosts some of the most well- 
known exerts in the world as well as nearly every mapr technology company. FreeBSD is 
proud to be a media partner for this years All Things Open. 
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